We currently use completely different methods and tools of building test beds and running tests for Debian vs.
Click packages, for normal uploads vs.
CI airline landings vs. upstream project merge proposal testing, and keep lots of knowledge about Click package test metadata external and not easily accessible/discoverable.
Today I released
autopkgtest 3.0 (and 3.0.1 with a few minor updates) which is a major milestone in unifying how we run package tests both locally and in production CI. The goals of this are:
- Keep all test metadata, such as test dependencies, commands to run the test etc., in the project/package source itself instead of external. We have had that for a long time for Debian packages with DEP-8 and
debian/tests/control, but not yet for Ubuntu s Click packages.
- Use the same tools for Debian and Click packages to simplify what developers have to know about and to reduce the amount of test infrastructure code to maintain.
- Use the exact same testbeds and test runners in production CI than what developers use locally, so that you can reproduce and investigate failures.
- Re-use the existing autopkgtest capabilities for using various kinds of testbeds, and conversely, making all new testbed types immediately available to all package formats.
- Stop putting tests into the Ubuntu archive as packages (such as
mediaplayer-app-autopilot). This just adds packaging and archive space overhead and also makes updating tests a lot harder and taking longer than it should.
So, let s dive into the new features!
New runner: adt-virt-ssh
We want to run tests on real hardware such as a laptop of a particular brand with a particular graphics card, or an Ubuntu phone. We also want to restructure our current CI machinery to run tests on a real OpenStack cloud and gradually get rid of our hand-maintained QA lab with its test machines. While these use cases seem rather different, they both have in common that there is an already existing machine which is pretty much only accessible with ssh. Once you have an ssh connection, they look pretty much the same, you just need different initial setup (like fiddling with adb, calling nova boot, etc.) to prepare them.
So the new
adt-virt-ssh runner factorizes all the common bits such as communicating with
adt-run, auto-detecting
sudo availability, doing SSH connection sharing etc., and delegates the target specific bits to a setup script . E. g. we could specify
--setup-script ssh-setup-nova or
--setup-script ssh-setup-adb which would then get called with
open at the appropriate time by
adt-run; it calls the nova commands to create a VM, or run a few adb commands to install/start ssh and install the public key. Then autopkgtest does its thing, and eventually calls the script with
cleanup again. The actual protocol is a bit more involved (see manpage), but that s the general idea.
autopkgtest now ships readymade scripts for these two use cases. So you could e. g. run the
libpng tests in a temporary cloud VM:
# if you don't have one, create it with "nova keypair-create"
$ nova keypair-list
[...]
pitti 9f:31:cf:78:50:4f:42:04:7a:87:d7:2a:75:5e:46:56
# find a suitable image
$ nova image-list
[...]
ca2e362c-62c9-4c0d-82a6-5d6a37fcb251 Ubuntu Server 14.04 LTS (amd64 20140607.1) - Partner Image ACTIVE
$ nova flavor-list
[...]
100 standard.xsmall 1024 10 10 1 1.0 N/A
# now run the tests: please be patient, this takes a few mins!
$ adt-run libpng --setup-commands="apt-get update" --- ssh -s /usr/share/autopkgtest/ssh-setup/nova -- \
-f standard.xsmall -i ca2e362c-62c9-4c0d-82a6-5d6a37fcb251 -k pitti
[...]
adt-run [16:23:16]: test build: - - - - - - - - - - results - - - - - - - - - -
build PASS
adt-run: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ tests done.
Please see
man adt-virt-ssh for details how to use it and how to write setup scripts. There is also a commented
/usr/share/autopkgtest/ssh-setup/SKELETON template for writing your own for your use cases. You can also not use any setup script and just specify user and host name as options, but please remember that the ssh runner
cannot clean up after itself, so never use this on important machines which you can t reset/reinstall!
Test dependency installation without apt/root
Ubuntu phones with system images have a read-only file system where you can t install test dependencies with apt. A similar case is using the null runner without root. When
apt-get install is not available, autopkgtest now has a reduced fallback mode: it downloads the required test dependencies, unpacks them into a temporary directory, and runs the tests with
$PATH,
$PYTHONPATH,
$GI_TYPELIB_PATH, etc. pointing to the unpacked temp dir. Of course this only works for packages which are relocatable in that way, i. e. libraries, Python modules, or command line tools; it will totally fail for things which look for config files, plugins etc. in hardcoded directory paths. But it s good enough for the purposes of Click package testing such as installing autopilot, libautopilot-qt etc.
Click package support
autopkgtest now recognizes click source directories and
*.click package arguments, and introduces a
new test metadata specification syntax in a
click package manifest. This is similar in spirit and capabilities to DEP-8
debian/tests/control, except that it s using JSON:
"x-test":
"unit": "tests/unittests",
"smoke":
"path": "tests/smoketest",
"depends": ["shunit2", "moreutils"],
"restrictions": ["allow-stderr"]
,
"another":
"command": "echo hello > /tmp/world.txt"
For convenience, there is also some magic to make running autopilot tests particularly simple. E. g. our existing click packages usually specify something like
"x-test":
"autopilot": "ubuntu_calculator_app"
which is enough to do what I mean , i. e. implicitly add the autopilot test depends and run autopilot with the specified test module name. You can specify your own dependencies and/or commands, and restrictions etc., of course.
So with this, and the previous support for non-apt test dependencies and the ssh runner, we can put all this together to run the tests for e. g. the Ubuntu calculator app on the phone:
$ bzr branch lp:ubuntu-calculator-app
# built straight from that branch; TODO: where is the official" download URL?
$ wget http://people.canonical.com/~pitti/tmp/com.ubuntu.calculator_1.3.283_all.click
$ adt-run ubuntu-calculator-app/ com.ubuntu.calculator_1.3.283_all.click --- \
ssh -s /usr/share/autopkgtest/ssh-setup/adb
[..]
Traceback (most recent call last):
File "/tmp/adt-run.KfY5bG/tree/tests/autopilot/ubuntu_calculator_app/tests/test_simple_page.py", line 93, in test_divide_with_infinity_length_result_number
self._assert_result("0.33333333")
File "/tmp/adt-run.KfY5bG/tree/tests/autopilot/ubuntu_calculator_app/tests/test_simple_page.py", line 63, in _assert_result
self.main_view.get_result, Eventually(Equals(expected_result)))
File "/usr/lib/python3/dist-packages/testtools/testcase.py", line 406, in assertThat
raise mismatch_error
testtools.matchers._impl.MismatchError: After 10.0 seconds test failed: '0.33333333' != '0.3'
Ran 33 tests in 295.586s
FAILED (failures=1)
Note that the current adb ssh setup script deals with some things like applying the autopilot click AppArmor hooks and disabling screen dimming, but it does not do the first-time setup (connecting to network, doing the gesture intro) and unlocking the screen. These are still on the TODO list, but I need to find out how to do these properly. Help appreciated!
Click app tests in schroot/containers
But, that s not the only thing you can do! autopkgtest has all these other runners, so why not try and run them in a schroot or container? To emulate the environment of an Ubuntu Touch session I wrote a
--setup-commands script:
adt-run --setup-commands /usr/share/autopkgtest/setup-commands/ubuntu-touch-session \
ubuntu-calculator-app/ com.ubuntu.calculator_1.3.283_all.click --- schroot utopic
This will actually work in the sense of running (and succeeding) the autopilot tests, but it will fail due to a lot of
libust[11345/11358]: Error: Error opening shm /lttng-ust-wait... warnings on stderr. I don t know what these mean, just that I also see them on the phone itself occasionally.
I also wrote another setup-commands script which emulates read-only apt , so that you can test the unpack only fallback. So you could prepare a container with click and the App framework preinstalled (so that it doesn t always take ages to install them), starting from a standard
adt-build-lxc container:
$ sudo lxc-clone -o adt-utopic -n click
$ sudo lxc-start -n click
# run "sudo apt-get install click ubuntu-sdk-libs ubuntu-app-launch-tools" there
# then "sudo powerdown"
# current apparmor profile doesn't allow remounting something read-only
$ echo "lxc.aa_profile = unconfined" sudo tee -a /var/lib/lxc/click/config
Now that container has enough stuff preinstalled to be reasonably fast to set up, and the remaining test dependencies (mostly autopilot) work fine with the unpack/
$*_PATH fallback:
$ adt-run --setup-commands /usr/share/autopkgtest/setup-commands/ubuntu-touch-session \
--setup-commands /usr/share/autopkgtest/setup-commands/ro-apt \
ubuntu-calculator-app/ com.ubuntu.calculator_1.3.283_all.click \
--- lxc -es click
This will successfully run all the tests, and provided you have apt-cacher-ng installed, it only takes a few seconds to set up. This might be a nice thing to do on merge proposals, if you don t have an actual phone at hand, or don t want to clutter it up.
autopkgtest 3.0.1 will be available in Utopic tomorrow (through autosyncs). If you can t wait to try it out,
download it from my people.c.c page .
Feedback appreciated!
There s a lot of discussion / moaning /arguing at this time, so I thought I d post something about how LAVA got into Debian Jessie, the work involved and the lessons I ve learnt. Hopefully, it will help someone avoid the disappointment of having their package missing the migration into a future stable release. This was going to be a talk at the Minidebconf-uk in Cambridge but I decided to put this out as a permanent blog entry in the hope that it will be a useful reference for the future, not just Jessie.
Context
LAVA relies on a number of dependencies which were at the time all this started NEW to Debian as well as many others already in Debian. I d been running LAVA using packages on my own system for a few months before the packages were ready for use on the main servers (I never actually installed LAVA using the old virtualenv method on my own systems, except in a VM). I did do quite a lot of this on my own but I also had a team supporting the effort and valuing the benefits of moving to a packaged system.
At the time, LAVA was based on Ubuntu (12.04 LTS Precise Pangolin) and a new Ubuntu LTS was close (Trusty Tahr 14.04) but I started work on this in 2013. By the time my packages were ready for general usage, it was winter 2013 and much too close to get anything into Ubuntu in time for Trusty. So I started a local repo using space provided by Linaro. At the same time, I started uploading the dependencies to Debian. json-schema-validator, django-testscenarios and others arrived in April and May 2014. (Trusty was released in April). LAVA arrived in NEW in May, being accepted into unstable at the end of June. LAVA arrived in testing for the first time in July 2014.
Upstream development continued apace and a regular monthly upload, with some hotfixes in between, continued until close to the freeze.
At this point, note that although upstream is a medium sized team, the Debian packaging also has a team but all the uploads were made by me. I planned ahead. I knew that I would be going to Macau for Linaro Connect in February a critical stage in the finalisation of the packages and the migration of existing instances from the old methods. I knew that I would be on vacation from August through to the end of September 2014 including at least two weeks with absolutely no connectivity of any kind.
Right at this time, Django1.7 arrived in experimental with the intent to go into unstable and hence into Jessie. This was a headache for me, I initially sought to delay the migration until after Jessie. However, we discussed it upstream, allocated time within the busy schedule and also sought help from within Debian with the RFH tag. Rapha l Hertzog contributed patches for django1.7 support and we worked on those patches upstream, once I was back from vacation. (The final week of my vacation was a work conference, so we had everyone together at one hacking table.)
Still there was more to do, the django1.7 patches allowed the unit tests to complete but broke other parts of the lava-server package and needed subsequent tweaks and fixes.
Even with all this, the auto-removal from testing for packages affected by RC bugs in their dependencies became very important to monitor (it still is). It would be useful if some packages had less complex dependency chains (I m looking at you, uwsgi) as the auto-removal also covers build-depends. This led to some more headaches with libmatheval. I m not good with functional programming languages, I did have some exposure to Scheme when working on Gnucash upstream but it wasn t pleasant. The thought of fixing a scheme problem in the test suite of libmatheval was daunting. Again though, asking for help, I found people in the upstream team who wanted to refresh their use of scheme and were able to help out. The fix migrated into testing in October.
Just for added complications, lava-server gained a few RC bugs of it s own during that time too fixed upstream but awkward nonetheless.
Achievement unlocked
So that s how a complex package like lava-server gets into stable. With a lot of help. The main problem with top-level packages like this is the sheer weight of the dependency chain. Something seemingly unrelated (like libmatheval) can seriously derail the migrations. The package doesn t use the matheval support provided by uwsgi. The bug in matheval wasn t in the parts of matheval used by uwsgi. It wasn t in a language I am at all comfortable in fixing but it s my name on the changelog of the NMU. That happened because I asked for help. OK, when django1.7 was scheduled to arrive in Debian unstable and I knew that lava was not ready, I reacted out of fear and anxiety. However, I sought help, help was provided and that help was enough to get upstream to a point where the side-effects of the required changes could be fixed.
Maintaining a top-level package in Debian is becoming more like maintaining a core package in Debian and that is a good thing. When your package has a lot of dependencies, those dependencies become part of the maintenance workload of your package. It doesn t matter if those are install time dependencies, build dependencies or reverse dependencies. It doesn t actually matter if the issues in those packages are in languages you would personally wish to be expunged from the archive. It becomes your problem but not yours alone.
Debian has a lot of flames right now and Enrico encouraged us to look at what else is actually happening in Debian besides those arguments. Well, on top of all this with lava, I also did what I could to help the arm64 port along and I m very happy that this has been accepted into Jessie as an official release architecture. That s a much bigger story than LAVA yet LAVA was and remains instrumental in how arm64 gained the support in the kernel and various upstreams which allowed patches to be accepted and fixes to be incorporated into Debian packages.
So a roll call of helpers who may otherwise not have been recognised via changelogs, in no particular order:
There is 
since people keep talking to me about my RC bug fixing activities, I thought
it might be time again for a short report. to be honest, I mostly stopped my
almost daily work at some point in december, partly because the overall
number of RC bugs affecting both